generalize first-write persistence and relax byte-zero read skipping #367
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… write It's relatively common that a bucket policy will remove ALL fragments of an idle journal. Later, the journal will suddenly have new writes, and in this case it's strongly desired for a new dirtying fragment write to occur as quickly as possible.
Seek forward a journal read at offset zero, to a first persisted fragment of the journal. This improves the ergonomics of a fairly common case where all data is deleted from an idle journal (for example, because of bucket lifecycle), and then new writes arrive, and a new reader is trying to read the journal from "the beginning". For this special case, we'd like reads to proceed forward as soon as a persisted fragment is available in the index without waiting for the full offsetJumpAgeThreshold.
jgraettinger
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Gazette has long had behavior of immediately persisting a fragment having Begin == 0. This has the effect of "dirtying" the fragment index, which ensures that other brokers understand the journal is not in a pristine state in the event of a loss of Etcd consistency.
Generalize this handling to immediately persist a Fragment if it's the first known write of the index, even if at an offset greater than zero. This better handles cases where a journal has been idle for a while, and all of its fragments have expired out via bucket policy, and suddenly new writes for the journal arrive.
Also slightly relax read offset skips to immediately skip-forward a read from (only) byte zero, to a first available persisted fragment. This lets such reads proceed much more quickly, rather than waiting for the six-hour delay that's used to guard against the possibility of raced reads vs writes to the fragment index during topology changes.
Testing:
This change is